Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup

Quantifying and comparing performance of numerical optimization algorithms is an important aspect of research in search and optimization. However, this task turns out to be tedious and difficult to realize even in the single-objective case – at least if one is willing to accomplish it in a scientifically decent and rigorous way. The COCO software used for the BBOB workshops (2009, 2010 and 2012) furnishes most of this tedious task for its participants: (1) choice and implementation of a wellmotivated single-objective benchmark function testbed, (2) design of an experimental set-up, (3) generation of data output for (4) post-processing and presentation of the results in graphs and tables. What remains to be done for practitioners is to allocate CPU-time, run their favorite black-box real-parameter optimizer in a few dimensions a few hundreds of times and execute the provided post-processing scripts. Two testbeds are provided, • noise-free functions • noisy functions and practitioners can freely choose any or all of them. The post-processing provides a quantitative performance assessment in graphs and tables, categorized by function properties like multi-modality, ill-conditioning, global structure, separability,. . . This document describes the experimental setup and touches the question of how the results are displayed. The benchmark function definitions, source code of the benchmark functions and for the post-processing and this report are available at http: //coco.gforge.inria.fr/. ∗NH is with the TAO Team of INRIA Saclay–Île-de-France at the LRI, Université-Paris Sud, 91405 Orsay cedex, France †AA is with the TAO Team of INRIA Saclay–Île-de-France at the LRI, Université-Paris Sud, 91405 Orsay cedex, France ‡SF is with the Research Center PPE, University of Applied Science Vorarlberg, Hochschulstrasse 1, 6850 Dornbirn, Austria §RR has been with the TAO Team of INRIA Saclay–Île-de-France at the LRI, UniversitéParis Sud, 91405 Orsay cedex, France